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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address- 
All claims being allowable, PROSECUTION ON THE MERITS IS (OR REMAINS) CLOSED in this application. If not included 
herewith (or previously mailed), a Notice of Allowance (PTOL-85) or other appropriate communication will be mailed in due course. THIS 
NOTICE OF ALLOWABILITY IS NOT A GRANT OF PATENT RIGHTS. This application is subject to withdrawal from issue at the initiative 
of the Office or upon petition by the applicant. See 37 CFR 1.313 and MPEP 1308. 

1 . ^ This communication is responsive to 05/13/2004 . 

2. £3 The allowed claim(s) is/are 1.5-14. 16-20:22-24.32-38 and 42 . 

3. ^ The drawings filed on 30 October 2000 are accepted by the Examiner. 

4. □ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 

a) □ All b) □ Some* c) □ None of the: 

1 . □ Certified copies of the priority documents have been received. 

2. □ Certified copies of the priority documents have been received in Application No. . 

3. □ Copies of the certified copies of the priority documents have been received in this national stage application from the 

International Bureau (PCT Rule 17.2(a)). 
* Certified copies not received: . 

Applicant has THREE MONTHS FROM THE "MAILING DATE" of this communication to file a reply complying with the requirements 
noted below. Failure to timely comply will result in ABANDONMENT of this application. 
THIS THREE-MONTH PERIOD IS NOT EXTENDABLE. 

5. □ A SUBSTITUTE OATH OR DECLARATION must be submitted. Note the attached EXAMINER'S AMENDMENT or NOTICE OF 

INFORMAL PATENT APPLICATION (PTO-1 52) which gives reason(s) why the oath or declaration is deficient. 

6. □ CORRECTED DRAWINGS ( as "replacement sheets") must be submitted. 

(a) □ including changes required by the Notice of Draftsperson's Patent Drawing Review ( PTO-948) attached 

1 ) □ hereto or 2) □ to Paper NoVMail Date . 

(b) □ including changes required by the attached Examiner's Amendment / Comment or in the Office action of 

Paper No./Mail Date . 

Identifying indicia such as the application number (see 37 CFR 1.84(c)) should be written on the drawings in the front (not the back) of 
each sheet. Replacement sheet(s) should be labeled as such in the header according to 37 CFR 1.121(d). 

7. □ DEPOSIT OF and/or INFORMATION about the deposit of BIOLOGICAL MATERIAL must be submitted. Note the 

attached Examiner's comment regarding REQUIREMENT FOR THE DEPOSIT OF BIOLOGICAL MATERIAL. 
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DETAILED ACTION 



1 . Claims 1-42 are in this application and claims 2-4, 15, 21, 25-31 and 39-41 are 
canceled by the examiner amendment. 1 , 5-14, 16-20, 22-24, 32-38 and 42 are 
pending this application. 



2. An examiner's amendment to the record appears below. Should the changes 
and/or additions be unacceptable to applicant, an amendment may be filed as provided 
by 37 CFR 1 .312. To ensure consideration of such an amendment, it MUST be 
submitted no later than the payment of the issue fee. 

Authorization for this examiner's amendment was given in a telephone interview 
with May on May 13, 2004. 



Please replace claim 1 with; --|1 . A method comprising: 

identifying, in response to a search query, first multimedia objects having an 
associated keyword that matches a keyword in the search query, and identifying second 
multimedia objects that have content features similar to those of the first multimedia 
objects, wherein the content features do not match the keyword in the search query; 
presenting the first and second multimedia objects to a user; 
monitoring feedback from the user as to which of the first and second multimedia 
objects are relevant to the search query; 



EXAMINER'S AMENDMENT 



Please cancefthese claims 2-4, 15, 21, 25-31 and 39-41. 
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maintaining associations between the keywords and the multimedia objects, the 
associations being weighted to indicate how relevant the keywords are to the 
multimedia objects; 

adjusting the weights of the associations based on the user's feedback, wherein 
the adjusting comprises increasing a weight of an association between the keyword and 
a particular multimedia object that is deemed relevant by the user or/and decreasing a 
weight of an association between the keyword and a particular multimedia object that is 
deemed irrelevant by the user; and 

annotating one or more of the second multimedia objects, which are deemed 
relevant by the user, with the keyword .-- 

r** : ■ 



Please cancel claim 2. 
Please cancel claim 3. 
Please cancel claim 4. 



f Please replace claim 10 with;- 



method comprising: 
identifying, in response to a search query, first multimedia objects having an 
associated keyword that matches a keyword in the search query, and identifying second 
multimedia objects that have content features similar to those of the first multimedia 
objects, wherein the content features do not match the keyword in the search query; 

presenting the first and second multimedia objects to a user; 
monitoring feedback from the user as to which of the first and second multimedia 
objects are relevant to the search query; 
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maintaining associations between the keywords and the multimedia objects, the 
associations being weighted to indicate how relevant the keywords are to the 
multimedia objects; 

adjusting the weights of the associations based on the user's feedback, wherein 
the adjusting comprises increasing a weight of an association between the keyword and 
a particular multimedia object that is deemed relevant by the user or/and decreasing a 
weight of an association between the keyword and a particular multimedia object that is 
deemed irrelevant by the user; and 

Annotating one or more of the second multimedia objects, which are deemed 
_relevant by the user, with the keyword.-- 

Please cancel claim 15. / 

Please replace claim 18 with; -Hff A method comprising: 
identifying, in response to a search query, first multimedia objects having an associated 
keyword that matches a keyword in the search query, and identifying second multimedia 
objects that have content features similar to those of the first multimedia objects, 
wherein the content features do not match the keyword in the search query; 

presenting the first and second multimedia objects to a user; 

monitoring feedback from the user as to which of the multimedia objects are 
relevant; 

annotating one or more of the multimedia objects based on the user's feedback, 
with a keyword; 



tfo 
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in an event that a particular multimedia object is deemed relevant by the user 
and not yet annotated with the keyword, adding the keyword to the particular multimedia 
object; and 

in an event that the particular multimedia object is deemed relevant by the user 
and is already annotated with the keyword, strengthening an association between the 
keyword and the particular multimedia object.-- 



Please cancel claim 21. 
Please cancel claim 25. 
Please cancel claim 26. 
Please cancel claim 27. 
Please cancel claim 28. 
Please cancel claim 29. 
Please cancel claim 30. 
Please cancel claim 31. 
Please replace claim 32 with 



A system comprising: 




an information retrieval unit to retrieve multimedia objects from a database based on a 
search query first multimedia objects having an associated keyword that matches a 
keyword in the search query, and identifying second multimedia objects that have 
content features similar to those of the first multimedia objects wherein the content 
features do not match the keyword in the search query; 

a relevance feedback unit to capture a user's feedback as to whether the 
multimedia objects are relevant to the search query; and 
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an annotation unit to annotate, with the keyword, the second multimedia objects 
based on the user's feedback, 

wherein the search query comprises a keyword-based search query having at 
least one keyword and in an event that a particular multimedia object is deemed 
relevant by the user and is already annotated with the keyword, the annotation unit 
strengthens an association between the keyword and the particular multimedia object, 
and/or in an event that a particular multimedia object is deemed irrelevant by the user 
and is already annotated with the keyword, removing the keyword from the particular 
multimedia object, and/or in an event that a particular multimedia object is deemed 
irrelevant by the user and is already annotated with the keyword, weakening an 

association between the keyword and the particular multimedia object. — 

Please cancel claim 39. 
Please cancel claim 40. 
Please cancel claim 41 . 

Allowable Subject Matter 

3. Claims 1 , 5-14, 1 6-20, 22-24, 32-38 and 42 are allowed over prior art made of 
record. 

The following is an examiner's statement of reasons for allowance: None of the 
non prior art alone or incombination neither teaches nor suggest "identifying, in 
response to a search query, first multimedia objects having an associated keyword that 
matches a keyword in the search query, and identifying second multimedia objects that 
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have content features similar to those of the first multimedia objects, wherein the 
content features do not match the keyword in the search query; 

presenting the first and second multimedia objects to a user; 

monitoring feedback from the user as to which of the first and second multimedia 
objects are relevant to the search query; 

maintaining, associations between the keywords and the multimedia objects, the 
associations being weighted to indicate how relevant the keywords are to the 
multimedia objects; 

adjusting the weights of the associations based on the user's feedback, wherein 
the adjusting comprises increasing a weight of an association between the keyword and 
a particular multimedia object that is deemed relevant by the user or/and decreasing a 
weight of an association between the keyword and a particular multimedia object that is 
deemed irrelevant by the user; and 

annotating one or more of the second multimedia objects, which are deemed 
relevant by the user, with the keyword." 

Any comments considered necessary by applicant must be submitted no later 
than the payment of the issue fee and, to avoid processing delays, should preferably 
accompany the issue fee. Such submissions should be clearly labeled "Comments on 
Statement of Reasons for Allowance." 



Conclusion 
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4. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Aggarwal et al. (US. Patent No. 6,728,706 B2) Patent date: 06/03/2003 



Shevade et al., "an experiental annotation system", Year 2003, ISBA:1-5081 13, 
page 91-98. 

Lai etal., "PBIR-MM: multimodal image retrieval and annotation", Year 2002, 
ISBN: 1-581 13-620-X, page 421-422. 



5. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Baoquoc N. To whose telephone number is (703) 305- 
1949 or via e-mail BaoquocN.To@uspto.gov. The examiner can normally be reached 
on Monday-Friday: 8:00 AM -4:30 PM, EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John Breene can be reached at (703) 305-9790. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305- 
3900. 

Any response to this action should be mailed to: 



Fox etal. (US. Patent No. 6,574,632 B2) 



Patent date: 03/03/2003 



Contact information 



Commissioner of Patents and Trademarks 
Washington, D.C. 20231. 



The fax numbers for the organization where this application or proceeding is 
assigned are as follow: 

(703) 872-9306 [Official Communication] 
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Hand-delivered responses should be brought to: 
Crystal Park II 
2121 Crystal Drive 
Arlington, VA 22202 
Fourth Floor (Receptionist). 

Baoquoc N. To 
May 13, 2004 
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